DPDK accelerated Tungsten Fabric dataplane

Since early 1990 network device manufacturer made a lot of innovation in order to increase router speeds. They started from a router node in which everything was computed into the central CPU to reach a situation where the central CPU is less and less used due to a distributed architecture in which lots of action are done in “line cards”.

These progresses have been made thanks to the use of proprietary TCAM (Ternary Content-Addressable Memory) and ASICs (Application-Specific Integrated Circuit) which have been designed to perform table look up and data packets forwarding at high speed.

In early 2000, the Virtualization for x86 computers support has led to lots of innovation into systems domain. Compute virtualization and High-Speed network devices evolution have enabled the Cloud creation.

Later, It appears it was not convenient to manage several isolated network devices each having their own configuration language. Following needs have emerged:

And these desires called for the cloud and SDN technology development.

In Stanford University (US - CA), Clean Slate Research Projects program has been initiated in order to think about how to improve the Internet network architecture. "ETHANE" project was part of this program. Its purpose was to " Design network where connectivity is governed by high-level, global policy". This project is generally known as the first implementation of SDN.

In 2008, a white paper has been proposed by ACM (Association for Computing Machinery) to design a new protocol (OpenFlow) that can program network devices from a network controller.

In 2011, ONF (Open Networking Foundation) has been created to promote SDN Architecture and OpenFlow protocols.

First companies working on SDN have been founded around 2010. (Most of them have now been acquired by main networks or virtualization solution vendors.) In 2007, Martin Casado, who was working on Ethane project has founded Nicira to provide solutions for network virtualization with SDN concept. Nicira has been aquired by vMware in 2012 to develop VMare NSX. In 2016, VMWare also bought PLUMGrid a SDN startup founded in 2013. In 2010, BigSwitch networks has been founded: BigSwitch is proposing a SDN solution. In early 2020, BigSwitch has been acquired by Arista Networks. In 2012, Cisco has created Insieme Networks, a spin-in start-up company working on SDN. In 2013, Cisco take back control on Insieme in order to develop its own SDN solution called ACI (Application Centric Infrastructure). In early 2012, Contrail Systems Inc has been created and aquired at the end of the year by Juniper Networks. In 2013, Alcatel Lucent has created Nuage Networks, a spin-in start-up company working on SDN. Nuage Networks is now an affiliate of Nokia.

The road of SDN development and its history is never straighforward and looks more nuanced than a single storyline might suggest. It’s actually far more complex to be described in a short section. This diagram from [sdn-history] shows developments in programmable networking over the past 20 years, and their chronological relationship to advances in network virtualization.

The concept of SDN, and the term itself, are both very broad and often confusing. There is no real accurate definition of SDN, and vendors usually take it very differently. Initially it was used to in Stanford’s OpenFlow project, and later it has been extended to include a much wider area of technologies. Discussion about each vendor’s exact SDN definition is beyond the scope of this book. But we generally consider that a SDN solution has to provide one to several of following characteristics:

According to [onf-sdn-definition], Software-Defined Networking (SDN) is:

In this diagram, you can see that SDN allows simple high-level policies in the "application layer" to modify the network, because the device level dependency is eliminated to some extent. The network administrator can operate the different vendor-specific devices in the "infrastructure layer" from a single software console - the "control layer". The "controller" in control layer is designed with such a way that it can view the whole network globally. This controller design helps a lot to introduce functionalities or programs as they just needs to talk to the centralized controller, without the need to know all details communicating with each individual device. These details are hidden by the controller from the applications.

Several expectations are behind this new model:

traditionally, A typical network device (e.g. a router) has following planes:

configuration and control plane are located in device’s main processor card, oftenly called "routing engine", or "routing switching engine". The forwarding plane is located in the device’s packet forwarding card, oftenly called "line card".

SDN architecture is built with 3 layers:

In a traditional infrastructure, the route calculation is made on each individual router. each router needs to run one or several routing protocols, through which it exchanges routes with the rest routers in the network, and eventually, based on the route information learned, each router assumes it gains enough knowledge about the network in order to make the forwarding decision. From the network perspective, the control plane is distributed in each individual router, and the end to end routing path is the result of all decisions made by the control plane located on each router.

The control plane on one router may look like this:

In reality, for example, a simplified Juniper MX control plane typical looks like this:

Running a control plane on each router make it very hard to manage, because each individual network device needs to be carefully configured. It requires extensive, vendor-specific experiences and skills to configure the device. The high number of configuration points often make it very challenging to build a robust network. Flexibility is also a recurring hurdle for traditional networks since most routers run proprietary hardware and software.

In contrast, in SDN networking, Control and Configuration functions are gathered into a "SDN controller" which is controlling Network devices. The new architecture intends to provide a completely new way to configure the network. This new Cloud infrastructure brings:

Let’s compare the two architectures:

This SDN infrastructure uses a centralized configuration and control point. route calculation is done centrally in the controller and distributed into each SDN network node. Well the idea looks good and simple, it requires a few foundamental protocols and infrastructures to be implemented before this model can work:

This underlay network infrastructure is playing the same role that the local switch fabric is doing inside a standalone router between the control processor card and lines cards. Based on it, A "overlay" network can be built by the controller, which basically hides underlay network infrastructure details from the applications so they will focus on the high level service implementations. we’ll talk more about "underlay" and "overlay" in the next section.

convenient as it is, this makes the controller the weakest point in the whole model. Think of what will happen if this SDN controller, serving as the "brain", stops working. Everything will be frozen and nothing works as expected, or even worse, some part of the infrastructure continues to run but in an unexpected way, which will very likely trigger bigger issues to other part of the network.

Lots of efforts are done by each SDN solution supplier to solve this weakness. A common and efficient practice is to use clustered architecture to build a highly resilient controller cluster. e.g 3 SDN controllers can load balance and/or backup each other. on failure of one or two, the other one can still make the whole cluster survive, giving the operator longer maintanence windows to fix the problem.

In SDN architecture, each network node is connected to a physical network infrastructure. This physical network which is providing basic connectivity between network nodes is called the "underlay" network infrastructure. sometimes it is also called "fabric", and typically it’s a plane L3 IP network.

very often The underlay needs to separate between different administrative domains (often called "tenants"), switch within the same L2 broadcast domain, route between L2 broadcast domains, provide IP separation via VRFs, and etc. This is implemented in the form of "overlay" networks. The overlay network is a logical network that runs on top of the underlay network. The overlay is formed of tunnels to carry the traffic across the L3 fabric.

Today the industry began to shift in the direction of building L3 data centers and L3 infrastructures, mostly due to the rich features coming from L3 technologies, e.g, ECMP load balancing, flooding control, etc. But the L2 traffic does not disappear and most likely it never will. There are always the desire that a group of network users need to reside in the same L2 network - typically a VLAN. However, In today’s virtualization environment, a user’s VM can be spawned in any compute located anywhere in the L3 cluster. Even if 2 VMs are spawned in the same server, there is often a need to move them around between different servers without changing their networking attributes. These requirements to make a VM always belonging to the "same VLAN" calls for an overlay model over the L3 network. In other words, we need a new mechanism to allow us to tunnel L2 Ethernet domains with different encapsulations over an L3 network.

For example, in SDN node1 we were running VM11 and VM12, they were both serving same sales department and so they were located in same VLAN. because of some administrative requirement, VM12 needs to be moved to another physical SDN node2 which, may be physically located in another rack that is a few router "hops" away. Now we need to ensure not only data packet from VM11 in SDN node1 to be able to reach VM12 in SDN node2, but also they are talking to each other as if they are still in the same VLAN, exactly the same way as before just as if VM12 has never moved. This ability to make the "local" (in same VLAN) traffic to traverse transparently across underlay network infrastructure calls for a packet encapsulation, or "tunneling" mechanism in SDN networks.

Indeed, without such an encapsulation mechanism, traditional segmentation solutions (VLAN, VRF) would have to be provided by the physical infrastructure and implemented up to each SDN node, in order to provide an isolated transportation channel for each customer network connected to the SDN infrastructure.

Encapsulation protocols used in SDN networks have to provide:

Several encapsulation protocols are used into SDN networks:

These encapsulation protocols are providing Overlay connectivity which is required between customers workload connected to the SDN infrastructure. Each SDN node is call a VTEP (Virtual Tunnel End Point) as it is starting and terminating the overlay tunnels.

We’ve seen "openflow" marked as one of the possible interfaces in the "SDN layer" section. Now we’ll introduce the concept of "southbound" and "northbound" interface and other available choices in today’s industry.

The "southbound" interface resides between the controller in "control layer" and network devices in "infrastructure layer". Basically what it does is to provide a means of communication between the 2 layers. Based on the demands and needs, a SDN Controller will dynamically changes the configuration or routing information of network devices. For example, a new VM will advertise a new subnet or host routes when it is spawned in a server, this advertisement will be delivered to SDN controller via a southbound protocol. Accordingly, SDN controller collects all routing updates from the whole SDN cluster through the southbound interfaces, and decides the most current and best route entries, then, it may "reflect" these information to all other network devices or VMs. This ensures all devices having the most uptodate routing information in real time. Among others, examples of the most well-known southbound interfaces in the industry are openflow, OVSDB and XMPP.

OpenFlow is one of the most widely deployed southbound standard from open source community. It first made its appearance in 2008 by Martin Casado at Stanford University. The appearance of OpenFlow was one of the main factors which gave birth to Software Defined Networking.

OpenFlow provides various information for the Controller. It generates the event-based messages in case of port or link changes. The protocol generates a flow based statistic for the forwarding network device and passes it to the controller.

OpenFlow also provides a rich set of protocol specifications for effective communication at the controller and switching element side. Open Flow provides an open source platform for Research Community.

Every physical or virtual OpenFlow-enabled network (data plane) devices in the SDN domain needs to first register with the OpenFlow controller. The registration process is completed via an OpenFlow HELLO packet originating from the OpenFlow device to the SDN controller.

unlike openflow, OVSDB is a southbound API designed to provide additional management or configuration capabilities like networking functions. With OVSDB we can create the virtual switch instances, set the interfaces and connect them to the switches. We can also provide the QoS policy for the interfaces.

The northbound interface provides connectivity between the controller and the network applications running in management plane. As we already discussed that southbound interface has OpenFlow as open source protocol, northbound lacks such type of protocol standards. However with the advancement of technology now we have a wide range of northbound API support like ad-hoc API’s, RESTful APIs etc. The selection of northbound interface usually depends on the programming language used in application development.

with the development of virtualization, SDN technologies and their ecology in recent years, more and more terms and changing of these terms emerge in the networking industry. a lot of confusions have rised, often because of terms are referring different things when they are used in different context. Sometimes the latest term the industry uses is a particular technology such as VNF or a concept such as NFV. Terms rise and fall out of favor as the industry evolves. In recent years the terms such as openstack, NVF/VNF has become the industry’s favorite buzzword. This raises the question - just what is openstack, NVF/VNF and what are the relationships with SDN?

NFV/VNF sounds like new buzzwords, but those technologies have been around for years. according to ETSI:

NFV means "network function virtualization", it stands for an "operation framework for orchestrating and automating VNFs". And VNF means "virtualized network function", such as virtualized routers, firewalls, load balancers, traffic optimizers, IDS or IPS, web application protectors, and so on.

in a nut shell you can think of NVF as a "concept", or "framework" to virtualize certain network functions, while VNF is the implementations of each individual network functions. Among others, firewalls and load balancers are the two most common VNFs in the industry, especially for deployments inside data centers. When you read today’s documents about virtualization technology, you will see the terms in such a pattern like "vXX" (e.g. vSRX, vMX), or "cXX" (e.g. cSRX) very often. that letter v indicates it is a "virtualized" product, while letter c - "containerized" is its container version.

Jointly launched by NASA and Rackspace in 2012, Openstack has rapidly gained popularity in many enterprise data centres. It is one of the most used open source cloud computing platform to support software development and Big Data analytics. OpenStack comprises a set of software modules, e.g, compute, storage & networking modules, which works together to provide an open source choice for building private & public cloud environments. As an IaaS (Infrastructure As A Service) open source implementation ,it provides a wide range of services, from basic service like computing service, storage service, networking service, etc, to advanced services like database, container orchestration and others.

You can think of Openstack as an abstraction layer providing a cloud environment on your promise. with openstack installed in your servers, ,you can spawn a VM, consume and recycle it when you are done, all in seconds. under that abstraction layer, Openstack hides most complexities of automation and orchestration of diverse underlying resources like compute, storage and networking. you could choose Servers, storage, networking devices from your favorite vendors to build the underlying infrastructure, and openstack will "consume" all of them and expose to the user as a pool of common "resources": number of CPUs, RAMs, hard disk spaces, IP addresses, etc. The user does not (need to) care about vendor and brand details.

If we compare openstack with SDN, it’s not hard to see that the two model shares some common features. Both models provide certain level of abstractions, hide the low level hardware details and expose to upper level user applications. the differences are somewhat subtle to describe in just a few words. First off, although there are various distributions from different vendors, they share common core components that is managed by the OpenStack Foundation. SDN is more of a "framework" or an "approach" to manage the network dynamically, which can be implemented with totally different software techniques. Secondly, From the perspective of technical ecological coverage, the ecological aspects of OpenStack are much wider, because networking is just one of its services that is implemented by its Neutron component among it’s other various plugins. SDN, and its ecology, in contrast, mainly focus on the networking. There are also difference in the way that Neutron works comparing with how a typical SDN controller works. OpenStack Neutron focuses on providing network services for virtual machines, containers, physical servers, etc, and provides a unified northbound REST API to users, SDN focuses on configuration and management of forwarding control toward the underlaying network device, it not only provides user-oriented northbound API, but also provides standard southbound API to communicating with various hardware devices.

As we’ve mentioned in previous sections, SDN is a networking scenario which changes the traditional network architecture by bringing all control functionalities to a single location and making centralized decisions. SDN controllers are the brain of SDN architecture, which perform the control decision tasks while routing the packets. Centralized decision capability for routing enhances the network performance. As a result, SDN controller is the core components of any SDN solutions.

While working with SDN architecture, one of the major point of concerns is which controller and solution should be selected for deployment. There are quite a few SDN controller and solutions implementations from various vendors, and every solution has its own pros and cons along with its working domain. In this section we’ll review some of the popular SDN controllers in the market, and the corresponding SDN solutions.

OpenDaylight, aften abbreviated as ODL, is a Java based open source project started from 2013, it was originally led by IBM and Cisco but later hosted under the Linux Foundation. it was the first open source Controller that can support non-OpenFlow southbound protocols, which can make it much easier to be integrated with multiple vendors.

ODL is a modular platform for SDN. It is not a single piece of software. It is a modular platform for integrating multiple plugins and modules under one umbrella There are many plugins and modules built for OpenDaylight. Some are in production, while some are still under development.

Some of the initial SDN controllers had their southbound APIs tightly bound to OpenFlow, But as we can see from the diagram, besides openflow, many other southbound protocols that are available in today’s market are also supported. Examples are NETCONF, OVSDB, SNMP, BGP, etc. Support of these protocols are done in a modular method in the form of different plugins, which are linked dynamically to a central component named "Service Abstraction Layer (SAL)". SAL does translations between the SDN application and the underlaying network equipments. for instance, when it receives a service request from a SDN application, typically via high level API calls (northbound), it understands the API call and translates the request to a language that the underlying network equipments can also understand. That language is one of the southbound protocols.

While this "translation" is transparent to the SDN application, ODL itself needs to know all the details about how to talk to each one of the network devices it supports, their features, capabilities etc. a topology manager module in OLD manages this type of information. What topology manager does is to collect topology related information from various modules and protocols, such as ARP, host tracker, device manager, switch manager, OpenFlow, etc, and based on these info, it visualize the network topology by drawing a diagram dynamically, all the managed devices and how they are connected together will be showed in it.

any topology changes, such as adding new devices, will be updated in the database and reflected immediately in the diagram.

Remember earlier we mentioned that an SDN controller has "global view" of the whole SDN network. In that sense ODL has all necessary visibility and knowledge of the network that can be used to draw the network diagram in realtime.

The Virtualized Cloud Platform (VCP) product from Nuage networks provides a highly scalable policy-based Software-Defined Networking (SDN) platform. It is an enterprise-grade offering that builds on top of the open source Open vSwitch for the data plane along with a feature-rich SDN controller built on open standards.

The Nuage platform uses overlays to provide seamless policy-based networking between Kubernetes Pods and non-Kubernetes environments (VMs and bare metal servers). Nuage’s policy abstraction model is designed with applications in mind and makes it easy to declare fine-grained policies for applications. The platform’s real-time analytics engine enables visibility and security monitoring for Kubernetes applications.

All VCS components can be installed in containers. There are no special hardware requirements.

The Tungsten Fabric (TF), is an open-standard based, proactive overlay SDN solution. It works with existing physical network devices and help address the networking challenges for self-service, automated, and vertically integrated cloud architecture. It also improves scalability through a proactive overlay virtual network technique.

TF controller integrates with most of the popular cloud management systems such as OpenStack, vmware, and Kubernetes. TF’s focus is to provide networking connectivity and functionalities, and enforce user-defined network and security policies to the various of workloads based on different platforms and orchestrators.

Tungsten Fabric’s primary claim to fame is that it is diligently multi-cloud and multi-stack. Today it supports:

TF fits seamlessly into LFN (Linux Foundation Networking) mission to foster open source innovation in the networking space.

The TF system is implemented as a set of nodes running on general-purpose x86 servers. Each node can be implemented as a separate physical server, or VM.

Initially, "Contrail" was a product of a startup company "Contrail system", which was acquired by Juniper Networks in Dec. 2012. It was open sourced in 2013 with a new name "OpenContrail" under the Apache 2.0 license, which means that anyone can use and modify the code of "Opencontrail" system without any obligation to publish or release the modifications. In early 2018, it was rebranded to "Tungsten Fabric" (abbreviated as "TF") as it transitioned into a fully-fledged Linux Foundation project. currently TF is still managed by the Linux Foundation.

Juniper also maintains a commercial version of the Contrail system, and provides commercial support to the payed users. Both The open-source version and commerical version of the Contrail system provide the same full functionalities, features and performances.

TF consists of two main components:

The communication between the controller and vRouters is via XMPP, which is a widely used messaging protocol.

A high level Tungsten Fabric architecture is shown below:

Kernel-based Virtual Machine (KVM) is an open source virtualization technology built into Linux. It provides hardware assist to the virtualization software, using built-in CPU virtualization technology to reduce virtualization overheads (cache, I/O, memory) and improving security.

QEMU is a hosted virtual machine emulator that provides a set of different hardware and device models for the guest machine. For the host, qemu appears as a regular process scheduled by the standard Linux scheduler, with its own process memory. In the process, QEMU allocates a memory region that the guest see as physical, and executes the virtual machine’s CPU instructions.

With KVM, QEMU can just create a virtual machine with virtual CPUs (vCPUs) that the processor is aware of, that runs native-speed instructions. When a special instruction is reached by KVM, like the ones that interacts with the devices or to special memory regions, vCPU pauses and informs QEMU of the cause of pause, allowing hypervisor to react to that event.

Virtio is an open specification for virtual machines' data I/O communication, offering a straightforward, efficient, standard and extensible mechanism for virtual devices, rather than boutique per-environment or per-OS mechanisms. It uses the fact that the guest can share memory with the host for I/O to implement that.

Virtio was developed as a standardized open interface for virtual machines (VMs) to access simplified devices such as block devices and network adaptors.

There are two parts to networking within VirtIO:

In Contrail Networking, VirtIO is used to connect with guest VM vNIC onto the vrouter vif interface.

In the diagram below you have a detailed view of a Virtual instance connectivity with a Kernel mode vrouter :

For more information on VirtIO :

Data Plane Development Kit (DPDK) is a set of data plane libraries and network interface controller drivers for fast packet processing, currently managed as an open-source project under the Linux Foundation.

The main goal of the DPDK is to provide a simple, complete framework for fast packet processing in data plane applications.

The framework creates a set of libraries for specific environments through the creation of an Environment Abstraction Layer (EAL), which may be specific to a mode of the Intel® architecture (32-bit or 64-bit), Linux* user space compilers or a specific platform.

These environments are created through the use of make files and configuration files. Once the EAL library is created, the user may link with the library to create their own applications.

The DPDK implements a "run to completion model" for packet processing, where all resources must be allocated prior to calling Data Plane applications, running as execution units on logical processing cores.

The model does not support a scheduler and all devices are accessed by polling. The primary reason for not using interrupts is the performance overhead imposed by interrupt processing.

For more information please refer to dpdk.org documents http://dpdk.org/doc/guides/prog_guide/index.html

NUMA means Non-Uniform Memory Access systems

A traditional server has a single CPU, a single RAM and a single RAM controller.

A RAM can be made of several DIMM banks in several sockets, all being associated to the CPU. When the CPU needs access to data in RAM, it requests it to its RAM controller.

Recent servers can have multiple CPUs, each one having its own RAM and its own RAM controller. Such systems are called NUMA systems, or Non-Uniform Memory Access. For example, in a server with 2 CPUs, each one can be a separate NUMA: NUMA0 and NUMA1.

When CPU0 needs to access data located in RAM0, it will go through its local RAM controller 0. Same thing happens for CPU1.

When CPU0 needs to access data located in the other RAM1, the first (local) controller 0 has to go through the second (or remote) RAM controller 1 which will access the (remote) data in RAM 1. Data will use an internal connection between the 2 CPUs called QPI, or Quick Path Interconnect, which is typically of a high enough capacity to avoid being a bottleneck, typically 1 or 2 times 25GBps (400 Gbps). For example the Intel Xeon E5 has 2 CPUs with 2 QPI links between them; Intel Xeon E7 has 4 CPUs, with a single QPI between pairs of CPUs.

The fastest RAM that the CPU has access to is the register, which is inside the CPU and reserved to it.

Beyond the register, the CPU has access to cached memory, which is a special memory based on higher performance hardware.

Cached memory are shared between the cores of a single CPU. Typical characteristics of memory cache are:

If the CPU needs to access data that is in the main RAM, it has to use its RAM controller.

Access to RAM takes then typically 170 CPU cycles (the green line in the diagram). Access to the remote RAM through the remote RAM controller typically adds 200 cycles (the red line in the diagram), meaning RAM latency is roughly doubled.

When data needed by the CPU is located both in the local and in the remote RAM with no particular structure, latency to access data can be unpredictable and unstable.

A single physical CPU core with hyper-threading appears as two logical CPUs to an operating system.

While the operating system sees two CPUs for each core, the actual CPU hardware only has a single set of execution resources for each core.

Hyper-threading allows the two logical CPU cores to share physical execution resources.

The sharing of resources allows two logical processors to work with each other more efficiently, and allows a logical processor to borrow resources from a stalled logical core (assuming both logical cores are associated with the same physical core). Hyper-threading can help speed processing up, but it’s nowhere near as good as having actual additional cores.

The performance of vRouter with sibling HT cores can increase by 10% to 20% (result is based on performance tests described hereinafter).

Memory is managed in blocks known as pages. On most systems, a page is 4Ki. 1Mi of memory is equal to 256 pages; 1Gi of memory is 256,000 pages, etc. CPUs have a built-in memory management unit that manages a list of these pages in hardware.

The Translation Lookaside Buffer (TLB) is a small hardware cache of virtual-to-physical page mappings. If the virtual address passed in a hardware instruction can be found in the TLB, the mapping can be determined quickly. If not, a TLB miss occurs, and the system falls back to slower, software based address translation. This results in performance issues. Since the size of the TLB is fixed, the only way to reduce the chance of a TLB miss is to increase the page size.

Virtual memory address lookup slows down when the number of entries increases.

A huge page is a memory page that is larger than 4Ki. In x86_64 architecture, in addition to standard 4KB memory page size, two larger page sizes are available: 2MB and 1GB.

Contrail DPDK vrouter can use both or only one huge page size.

Use lstopo (hwloc-nox package)and numactl commands to check cores numbering and numa nodes architecture

To check NIC assignment to NUMA node.

Use lspci command to find PCI addresses of NICs that will be bound to DPDK process

Use dpdk_nic_bind.py command to verify proper binding of NICs to DPDK

When planning physical CPU cores allocation, the following aspects need to be taken into account:

If HT (Hyper Threading) is enabled, the first half of each numa node core are physical cores, and the second half their HT sibling.

The best way to utilize all cores in the system, especially when using hw:cpu_policy=dedicated (see CPU isolation mechanisms) is to use both siblings when calculating mask for vrouter (and other resources line systemd). In that case, both siblings for each core used for VM pinning can be utilized.

With OpenStack, Having a core without its sibling listed in vcpu_pin_set variable in nova.conf file and hw:cpu_policy=dedicated set in flavor properties used to spawn an instance leads to a scheduling error (variable sibling_set referenced before assignment).

In order to get a CPU booked for a subset of tasks, we have to inform the Operating System scheduler not to use these CPUs for all the tasks it has to run.

These CPUs are told: "isolated" because they are no more used to process all tasks. In order to get a CPU isolated several mechanisms can be used:

RedHat Linux is currently supporting 2 different CPU isolation methods. They are isolcpus and tuned CPU partitioning, They can be used either independently or simultaneously.

A first method, isolcpus kernel parameter, has been proposed to keep CPUs away from linux scheduler. This isolation mechanism will:

The main drawbacks of using isolcpus are :

A second method, tuned CPU partitioning, has been proposed more recently. since almost all processes are started by systemd, physical CPU cores assignment can be configured in a different way:

This isolation mechanism will:

The main drawbacks of using tuned partitioning are :

In few words, tuned partitioning is more flexible and featureful than isolcpus, but is providing a lower CPU isolation. This is why both mechanisms are often used altogether to enforce CPU isolation.

DPDK control threads can be pinned on any CPU. These CPUs are only used during Contrail vRouter startup for DPDK initialization. As they did not require high CPU capacity, a good strategy is to use the CPU allocated to the Operating System for these threads.

Besides forwarding threads Contrail DPDK vRouter runs service threads :

Service and Processing threads are named lcore-slave-<ID>. Contrail vRouter cores ID have a specific meaning defined in the following C enum data structure, An enumeration is defining this numbering :

0 to 9 lcore numbers are statically defined into the source code and can’t be configured by user.

contrail and service cores should be pinned to dedicated core(s) if flow mode is heavly used, in packet mode can be use the CPU allocated to the Operating System. This value is set into /etc/sysconfig/network-scripts/ifcfg-vhost0 configuration file.

The value can be provided as list (in the example) or hex mask (0x540000054).

lcore numbers 10 and above are used for forwarding purpose and defined with CPU affinity value.

This value is set into /etc/sysconfig/network-scripts/ifcfg-vhost0 configuration file.

For instance 2,4,6,30,32,34 CPU list is giving following mapping :

it means Host CPU numbers 2, 4, 6, 30, 32 and 34 are used for vrouter forwarding logical cores.

When the router boots up, it displays a message about its logical cores numbering. For instance, we can get into contrail-vrouter-dpdk-stdout.log file, such a message :

This is this vrouter logical core numbering that has to be in dropstats command (not the real host CPU numbering)

A maximum of 16 polling cores is currently supported by Contrail as the maximum RX queue is currently 16:

We can find those names by running using the "ps" command with some additional arguments (this is from a compute node where vrouter is using 8 logical forwarding cores, 4 phy cores using HT siblings):

Here we have :

To list all lightweight processes created by contrail-vrouter-dpdk run following command:

The following command can be used. It provides in the last column the CPU load generated by each thread:

Using pidstat command we can see that some vrouter threads are running at 100% CPU. Those are the forwarding threads, the ones pinned based on the configured coremask :

In the example above, we can notice:

In order to list CPU cores assigned to contrail-vrouter-dpdk, we have to run taskset command for each lightweight process.

The packet arrives at a physical NIC (pNIC) The pNIC computes a hash on the packet outer header

Physical NIC (usually an interface bond) itself spread packets across its queues using 5-tuple hashing function (source IP, destination IP, source port, destination port and protocol).

Depending on overlay encapsulation protocol used, the incoming traffic is well balanced or not onto NIC card RX queues :

That is the reason Juniper recommends implementing MPLSoUDP on Juniper MXes (supported in Junos >=16.2).

Based on the "RSS hash", the pNIC selects a queue and writes the packet to the queue (RX QUEUE into the previous diagram).

Then incoming packet is dequeued by the polling logical core (step 1) :

Hashing algorithm is in vr_dpdk_ethdev.c module (https://github.com/Juniper/contrail-vrouter/blob/R5.0/dpdk/vr_dpdk_ethdev.c#L873)

In previous stage polling CPU bound to the queue on which a packet has been received, is triggering packet transfer from given NIC RX queue to mbuf.

This CPU (here CPU 0) will also choose which CPU (forwarding CPU) will have to process the packet (decapsulation, routing, switching, etc).

So a hash will be calculated onto polling CPU on decapsulated packet in order to select a forwarding CPU (decapsulated packet should have a better entropy than encapsulated one with MPLS over GRE).

Then the forwarding logical core is processing the packet (step 2). Forwarding CPU is in charge to do :

In the diagram below it is shown a situation where the polling CPU core is selecting another CPU for packet processing.

This would be the case for MPLSoGRE traffic, since the hash performed by the NIC is likely not efficient :

In previous stage polling CPU bound to the queue on which a packet has been received, is triggering packet transfer from given NIC RX queue to mbuf.

This CPU (here CPU 0) will also choose which CPU (forwarding CPU) will have to process the packet (decapsulation, routing, switching, etc).

But as incoming encapsulated packets are using UDP protocol, a good entropy is expected for incoming encapsulated traffic. No hash is recalculated on the polling core, and each incoming packet will be processed on the same forwarding core than the polling one.

In the diagram below, it is shown a situation where the polling CPU core is the same as the forwarding CPU, which will be the case for MPLSoUDP and VXLAN packets from the NIC:

When MPLS over UDP is used, there is internal traffic load balancing onto vrouter CPU. Incoming packets are processed with the same forwarding core as polling core :

It can be easily shown using a traffic generator to send a "single UDP" flow onto a virtual instance.

When single UDP flow is reaching the vrouter, RX port packets and RX packets counters on vif 0/0 have the same values for a same lcore: Here the VM incoming traffic is polled and processed by core 12:

Table above has been made using real single flow traffic seen on each vif 0/0 and vif 0/3 interface (vNIC) with vif --get command (cf appendixes)

vNIC queues are assigned to logical cores in the following way:

Log messages showing this process can be seen in the /var/log/containers/contrail/contrail-vrouter-dpdk.log file.

When interface is added and the queue to forwarding core mappings are made. There are six forwarding cores in this setup. Only the first queue is actually enabled at this point.

The remaining queues will use the mentioned forwarding cores if the VM chooses to enable them (i.e. it supports multiqueue).

When a VM (in this case a vSRX) requests to enable some of its available queues. The vrings here correspond to both transmit and receive queues.

The receive queues are the odd numbers. Divide them by 2 (discard the remainder) to get the queue number. i.e. vring 1 is queue 0. ready state 1 = enabled. ready state 0 = disabled. In this example, the vSRX is only enabling 4 queues:

RSS hashing depends on the packet type:

When polling CPU core (in the above example CPU core 3) polls the packet from VM virtio interface queue it selects which forwarding CPU core will process the incoming packet. This forwarding CPU core does flow lookup, flow enforcement, encapsulation and packet delivery to NIC transmit queue.

Then forwarding logical core is processing the packet (step 2). Forwarding CPU is in charge to do :

For VM incoming traffic a hash is calculated in order to rebalance received traffic on a given polling core to all other forwarding cores :

It can be easily shown using a traffic generator to send a "single UDP" flow onto a virtual instance.

When single UDP flow is reaching the vrouter, RX port and RX packets counters on vif 0/N have the same values for distinct lcores.

Here the VM incoming traffic is polled by core 13 and processed by core 11:

Table above has been made using real single flow traffic seen on each vif 0/0 and vif 0/3 interface (vNIC) with vif --get command (cf appendixes)

regarding RX port and RX packet, see Packet flow from a VM NIC to compute node Physical NIC - counters placement

The network performance does not scale as the number of vCPUs increases.

Guest cannot transmit or retrieve packets in parallel as virtio-net have only one TX and RX. virtio-net drivers must be synchronized before sending and receiving packets.

It is useless to assign more CPU to vrouter in order to increase packet processing rate, as only one vrouter CPU is used to poll the single vNIC queue.

Moreover, all vrouter CPU could have a packet to transmit to the vNIC single queue which is becoming a congestion point.

If we add more CPU, we increase the pressure on the vNIC single queue and it could lead to jitter and even packet drop (buffer overflow).

When multiqueue is used on a Guest VM, In the ideal scenario, each multiqueue vNIC is configured with the same number of queue than the number of CPU allocated to vrouter for packet processing, because the vrouter will only be able to sustain guest vNIC configured with a number of queue below or equal to the number for processing core allocated to the vrouter. The difference with non-DPDK is that vRouter runs in the user space instead of host kernel.

Also there is a vhost userspace process wherein the vRouter copies packets to/from the VM.

Every queue inside the VM maps to a virtio ring on the host user space.

Some of the offload capabilities, e.g. GRO (Generic Receive Offload), is not supported with multiqueue virtio on DPDK vrouter in Contrail 3.2.

In this case vRouter runs as a multi-core process that exists on all assigned cores.

There is also a scale-out approach to packet processing using multiple cores so the performance of one VM is NOT limited by the performance of 1 core.

A vRouter might have more cores than there are queues in the VM. Queues can only be shared when the vRouter cores send to the VM queue.

When receiving from the queue, exactly one vRouter core will read from a queue (i.e. receive queues are not shared, in order to prevent packet reordering).

vNIC queues are not configured by Contrail vRouter, they are managed by OpenStack. Rqueue Size are defined at OpenStack level (using libvirt) : /etc/nova/nova.conf

Nova configuration file values are provisioned with TripleO.

vNIC queues size can be defined using NovaLibvirtRxQueueSize and NovaLibvirtRxQueueSize environment parameters. By default they are configured with a 512 value.

Nova configuration file is configured with <tripleo-templates-root-dir>/puppet/services/nova-compute.yaml file:

To enable multiqueue with DPDK in a Glance image, we add metadata hw_vif_multiqueue_enabled="true":

Number of queues to be created can’t be specified with Glance metadata. Currently, the number of queues will match the number of vCPUs, defined for the instance.

Nova is configuring automatically the number of queues of each vNIC. There is no way to manually define the number of queues to be created for each vNIC. We can only change the number of queues of a given vNIC after the virtual instance has been started, using some Linux commande into the virtual machine whose vNIC configuration has to be modified (cf next section).

Nova is using vif.py script to create instance virtual interfaces (https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py)

Number of configured queues for a vif is depending on:

When a Linux Kernel 3.x is used (RHOSP 13), a maximum of 8 queues are configured by Nova for each vif.

When a Linux Kernel 4.x is used (RHOSP 16), the same algorithm is used but using 256 instead of 8 as a maximum of queues to be configured by each vif.

Inside an guest instantiated guest with multiqueues enabled, the NIC channel setup can be checked and changed as needed with the commands below, inside each virtual machine.

In order to see the current number of queues we have to use :

In order to change the number of queues we have to use :

For instance, below we are configuring 4 logical queues on eth0 vNIC :

This should be done during interface initialization, for example in a "pre-up" action in /etc/network/interfaces. But it seems to be possible to change this configuration on a running interface without disruption.

Using virsh dumpxml command, we can verify queue size configured by Nova with libvirt :

Here, 5 queues have been allocated to the vNIC, rx/tx_queue_size is configured as 1024.

The following kernel parameters configure huge pages:

To finish configuration of hugepages, hugetlbfs pseudo filesystem needs to be mounted. The following line needs to be added to /etc/fstab:

After adding the configuration to a kernel and rebooting server, the number of allocated huge pages can be checked from command line. There are multiple ways of doing this, The most common way of doing it is to print /proc/meminfo. This is not a good way, because it shows only one (default) huge page size. Since it is possible to configure multiple sizes of hugepages, the only source of information is /sys filesystem (for each numa node and each size - total/free):

As mentioned before, DPDK vRouter needs its own hugepages. The DPDK library auto-detects the huge pages from the hugetlbfs mount point.

2GB for vRouter are based on setup provided in configuration file.

In order to allocate memory on the first NUMA socket, we are using option with only one parameter:

In order to allocate memory on the NUMA0 and NUMA1 socket, we are using option with only two parameters:

On a 2 NUMA nodes we have to allocate memory on each of them even if vRouter is only pinned on a single one. Indeed Virtual machines are pinned on both and require memory on each. It is important to allocate hugepage memory to all NUMA nodes that will have DPDK interfaces associated with them. If memory is not allocated on a NUMA node associated with a physical NIC or VM, they cannot be used.

We can get all PCI Ethernet devices installed on the host using the following command :

We can check the PCI device related numa node id using one of following commands:

Using numactl we can get CPU IDs on each NUMA socket:

If only 1G huge pages are configured on the compute node, pagesize parameter must be specified into huge page table mount point in order for the vrouter to be aware that only 1G huge pages are available on the compute node:

Cf: https://github.com/Juniper/contrail-vrouter/blob/R1908/dpdk/vr_dpdk_table_mem.c#L80-L84

If this parameter is not present in the mountpoint, Contrail vrouter is assuming that 2MB hugepages are available, and will requests 2M huge pages.

If only 1GB huge pages are available onto the compute node, vrouter will fail to start.

When huge pages mount point is configured without pagesize parameter, vrouter will request 2M hugepages allocation.

Here we have specified huge pages mount with pagesize=1G parameter. vrouter will request 1G hugepages allocation at startup.

ping? talk about IRQ?

Disabling IRQ Balance is not recommended as it has no impact on performance. In RHEL, tuned’s cpu-partitioning profile leverages IRQ CPU affinity (isolated_cores).

stop irqbalance daemon and remove irqbalance from system startup:

The /etc/sysconfig/irqbalance configuration file contains a parameter named IRQBALANCE_BANNED_CPUS that allows to define CPUs to be excluded from consideration by the IRQ balancing service.

The following mask excludes CPUs 8 to 15 as well as CPU 33 from IRQ balancing:

(*) these features are requiring "mergeable buffers" to be supported.

ping? talk about mergeable buffers?

Intel recommendation to reduce or avoid packet loss

When using Niantic (PF), the packets can be either buffered in the RX descriptors (or, more correctly speaking, in the memory addressed by the RX descriptors), or in the RX packet buffer in the NIC.

RX packet buffer size is 512KB when flow director is disabled, hence it can hold > 8000 packets of 64 Bytes ⇒ handle an interrupt longer than 500 microseconds - https://etherealmind.com/wp-content/uploads/2017/01/X520_to_XL710_Tuning_The_Buffers.pdf )

When using Fortville, packets are by default dropped when RX descriptors are not available. Hence, by default, the only buffers available to store packets when a core is interrupted are the buffers pointed to by the RX descriptors.

A similar behavior (packets dropped when no descriptors available) is obtained on Niantic when using Virtual functions: to avoid "head of line" blocking, packets are dropped if no descriptor is available and virtual functions are used.

So, to avoid packet losses due to CPU core being interrupted when using Fortville (or when using Niantic and *SR-IOV*), the number of RX descriptors should be configured high enough, for instance to 2048.

Setup number of rx descriptors through rte_eth_rx_queue_setup (see http://www.dpdk.org/browse/dpdk/tree/examples/l3fwd/main.c#n990

The number of TX descriptors should also be considered and depends on the application. Imagine the CPU core is interrupted and up to 2048 packets are now buffered through the RX descriptors. Once the interrupt is dealt with, up to 2048 packets can be received very fast by the application: as the packets are already in memory, they might arrive much faster than 10Gbps. Hence, when the application tries to forward those packets, the application might try to send those packets faster than 10Gbps. Some of those packets must be buffered before being sent to the 10 Gb/s link. Either the application takes care of this by buffering those packets, or those packets must be buffered in the TX descriptors. In that case, the number of TX descriptors should also be increased (e.g. up to 2048).

Increasing the number of RX and/or TX descriptors implies that you might have to increase the number of pre-allocated mbufs. This can have a small impact on performance (throughput), as more memory is being used.

Setup number of mbufs through rte_pktmbuf_pool_create: (see http://www.dpdk.org/browse/dpdk/tree/examples/l3fwd/main.c#n724)

A problem with reassigning kernel driver back to interface when DPDK vRouter is stopped.

On Compute node In file /opt/contrail/bin/vrouter-functions.sh (line 532) in s/\x1([0-9:.]+)[\x1]+/ \1/g there are missing letters from a to f. This should be corrected and that could impact on not binding interface back to interface kernel driver from UIO kernel driver when vRouter is stopped.

The correct sed should have letters s/\x1([0-9a-f:\.]+)[^\x1]+/ \1/g.

This issue will be solved in release 5.0.1 https://bugs.launchpad.net/juniperopenstack/+bug/1774403

vRouter CPU_LIST (also named coremask in some configuration files) variable is allowing to define which core CPU will be allocated for router forwarding and polling processing threads.

vrouter forwarding and polling processings are CPU intensive. vrouter logical threads named logica core ID 10 to logical core 10 + N - 1 (with N = Number of allocated polling and forwarding cores) are used for forwarding and polling processing. These threads have to be pinned to some well defined CPU in order to avoid the vrouter to starve the full node CPU capacity. A maximum of 54 forwarding CPU can be allocated to a vrouter (logical core ID: 10 to 63).

Below is an example of a configuration of a 2 socket system, each processor with 2*18 physical cores, with HT enabled. The network adapter in PCI-E bus is attached to NUMA 0. 4 physical cores are dedicated for vRouter (i.e. 8 logical cores thanks to Hyper Threading).

We are planning CPU assignment as follows:

For optimal performance it is recommended that a single VMs is executed on a same NUMA (unless it can efficiently leverage kvm/pinning information)

vRouter CPU_LIST (aka coremask) calculation based on the assumption above: vCPUs 2, 4, 6, 8, 38, 40, 42, 44 are allocated for vRouter, which maps to b0001 0101 0100 0000 0000 0000 0000 0000 0000 0001 0101 0100 in binary.

This translates to a hex mask 0x154000000154 that must be configured in the /etc/contrail/supervisord_vrouter_files/contrail-vrouter-dpdk.ini file (see below example for Contrail 4.1 release).

After modifying this file, the vrouter must be restarted to take changes into account:

After modifying this file, the vrouter vhost0 interface must be restarted to take changes into account:

For contrail release 19.12 and earlier, Control and Service threads are not assigned to any CPU. Consequently, these threads can use any available CPU.

We can get all contrail-vrouter-dpdk assignments using taskset with -a option :

In the previous example, we can notice that threads are assigned to a given CPU set according following rules :

SERVICE_CORE_LIST has been split into two variables into Contrail 20.03 and later releases:

In order to be used, you’ll have to follow explanation in next section given for SERVICE_CORE_MASK (just replace in you configuration SERVICE_CORE_MASK with SERVICE_CORE_LIST when using this patch)

Since Contrail 20.03 release it is possible to assign Control and Service threads to a given CPU list. vRouter SERVICE_CORE_MASK variable is allowing to define which core CPU will be allocated for vrouter service threads. DPDK_CTRL_THREAD_MASK variable is allowing to define which core CPU will be allocated for vrouter control threads.

Below is an example of a configuration of a 2 socket system, each processor with 2*18 physical cores, with HT enabled.

The network adapter in PCI-E bus is attached to NUMA 0. 4 physical cores are dedicated for vRouter (i.e. 8 logical cores thanks to Hyper Threading).

We are planning CPU assignment as follows:

For optimal performance it is recommended that each VM be executed on a same NUMA (unless it can efficiently leverage kvm/pinning information).

vRouter service cores mask calculation based on the assumption above: vCPUs 10, 46 are allocated for vRouter, which maps to b0100 0000 0000 0000 0000 0000 0000 0000 0000 0100 0000 0000 in binary.

This translates to a hex mask 0x400000000400 that must be configured into SERVICE_CORE_MASK variable in /etc/sysconfig/network-scripts/ifcfg-vhost0 file:

vRouter DPDK control threads mask calculation based on the assumption above: vCPUs 0,1,3,5,36,37,39,41 are allocated for vRouter, which maps to binary:

This translates to a hex mask 0x2B00000002B that must be configured into DPDK_CTRL_TREAD_MASK variable in /etc/sysconfig/network-scripts/ifcfg-vhost0 file: DPDK_CTRL_THREAD_MASK=0x2B00000002B

After file has been updated, the vrouter vhost0 interface must be restarted to take changes into account:

We can get all contrail-vrouter-dpdk assignments using taskset with -a option :

In the previous example, we can notice that threads are assigned to a given CPU set according following rules :

Currently only CPU assigned to packet processing threads (CPU_LIST / COREMASK) can be managed with provisioning tools.

Since version 20.03, we can also manage CPU assigned to control and service threads (DPDK_CTRL_THREAD_MASK and SERVICE_CORE_MASK) with provisioning tools.

Contrail can be installed through different tools: Contrail Cloud, Contrail Networking integrated with RHOSP and TripleO deployer or Juniper Contrail Ansible Deployer to name some.

In non-multiqueue VNF with one DPDK interface, increasing the number of CPU cores for vRouter can actually decrease performance because of One to Many CPU cores queues are sending packets one by one what takes time and can produce microbursts.

On the diagram cores one by one are sending traffic to VM and when last finish data processing potentially first will have no time (buffers) to send VM (virtio queue).

During this case microbursts affecting VNF will be observed on vRouter as TX errors.

First we are enforcing CPU isolation rules in order to keep CPUs assigned to vRouter and Nova out of the system scheduler :

On Ubuntu OS, isolcpus mechanism is used in order to keep CPUs assigned to vRouter and Nova out of the system scheduler :

Cores to be used by Nova to run VMs is defined into vcpu_pin_set variable (in DEFAULT section in /etc/nova/nova.conf file).

The advantage of this approach is that for flavors without the hw:cpu_policy=dedicated property (the default value will be set hw:cpu_policy=prefered), VMs will be balanced between cpus.

With isolcpus set, all VMs without the above property will end up executing on the first cpu from the list.

In order to check actual CPU used to run a process and the CPU pinning rule bound to each process, following command can be used :

In the first column we can read the CPU pinning put onto the process with taskset command, in the second one (PSR), we can see the actual CPU used by the process.

The default flow table size is 512K and can be increased by setting the -– vr_flow_entries in /etc/contrail/supervisor-vrouter_files/contrail-vrouter-dpdk.ini

(Since Contrail 5.0 release and later, Contrail is now containerized and DPDK parameters are no more defined into contrail-vrouter-dpdk.ini file but are installed by /bin/bash /entrypoint.sh into contrail-vrouter-agent-dpdk container - cf appendix section)

To increase the flow table to 2M flows set the –vr_flow_entries as below:

Then reload kernel module and restart vrouter

For optimal flow setup rate, the agent flow thread count can be increased to 4 (default 2 threads).

Then run service supervisor-vrouter restart for the changes to take effect.

This setting is also available at orchestration level through the following variable in TripleO template:

To set about the total amount of threads assigned to the agent (not only for flows), the number of threads for agent should be higher than for the flow thread number.

Then run service supervisor-vrouter restart for the changes to take effect.

By default Contrail works in flow-based processing providing a vast feature portfolio. Alternatively, packet based mode can be enabled in Contrail on per interface basis, in order to increase the packet processing rate for traffic that may not need a processing per flow.

Packet-based processing is configurable at port level by selecting the "Disable Policy" checkbox: this will stop the creation of new flows for this interface only.

Despite disabling policy the following features will continue to be available:

During provisioning of Contrail sometimes provisioning tool not set an encapsulation priority by this for communication to gateways and among vrouters is using default MPLSoGRE. MPLSoGRE is less efficient and not support 5 tuple entropy. To change it encapsulation priority need to be set as follow:

The encapsulation can be set by WebUI.

In parallel, for optimal load balancing capabilities and resource usage optimisation the SDN gateway must also be configured with the MPLSoUDP encapsulation.

The below configuration applies for JUNOS devices (MX). Note that Junos MPLSoUDP support starts with 16.2 onwards.

In parallel, some VNF design can also require the activation of Load Balancing (a.ka. multipath) at VRF level.